Future (eventually) failed: addFlow -and- transmitPacket
Description
duplicates
Activity
Gobinath Suganthan July 12, 2018 at 7:24 AM
@Anil VishnoiWe don't use the lldp module and I'm not aware of the design. But my understanding is that the periodic thread will run after initialization but only the node connectors available when the node initially connected would be sending LLDP packets. In case of a newly added node connectors, we have to update the monitoring to include the new node connector too. In the existing implementation they explicitly want to send the lldp packets from the newly added node connectors before the scheduled thread pitches in. If there is no need for immediately detecting the new nodeconnector in the topology, we could do away with sending packets in nodeConnectorAdded() method.
Anil Vishnoi July 10, 2018 at 7:44 PM
@Gobinath Suganthan I think your patch looks good, but i think we should not send the packet in nodeConnectorAdded() method, because run() method is already doing that for you. So can you update your patch to get rid of the trasmitPacket() part of the code?
Anil Vishnoi July 10, 2018 at 7:28 PM
@Sam Hague Either should work. If you disable inactivity probe from switch side, controller will send the echo request to check the connection aliveness, but that's more safe because that goes according the controller state at that point of time. But for production environment it's not recommended to disable the inactivity_probe, because there is a possibility that connection from switch to controller can go in bad state (like you drop packet going from switch to controller, but other way around it's working fine).
Sam Hague July 10, 2018 at 1:12 PM
@Anil Vishnoi in the ofp csit, the inactivity_probe is set to 0 and the comment is to disable the echos. Should we disable it also or use the 30s?
Comment Below line disables switch echos
Write sh x=`sudo ovs-vsctl --columns=_uuid list Controller | awk '{print $NF}'`; for i in $x; do sudo ovs-vsctl set Controller $i inactivity_probe=0; doneSam Hague July 10, 2018 at 1:06 PM
We do not currently set it in netvirt CSIT. The value can be set using the elan-config.xml as shown below. Above @Anil Vishnoi showed how the ofp csit sets the value directly on the switches using ovs-vsctl. I can add patch to do the below for netvirt clustering csit.
<elanmanager-config xmlns="urn:opendaylight:netvirt:elan:config">
<auto-create-bridge>true</auto-create-bridge>
<int-bridge-gen-mac>true</int-bridge-gen-mac>
<temp-smac-learn-timeout>10</temp-smac-learn-timeout>
<arp-punt-timeout>5</arp-punt-timeout>
<punt-lldp-to-controller>false</punt-lldp-to-controller>
<!--
<controller-max-backoff>5000</controller-max-backoff>
<controller-inactivity-probe>5000</controller-inactivity-probe>
-->
<auto-config-transport-zones>false</auto-config-transport-zones>
<use-of-tunnels>false</use-of-tunnels>
<openstack-vni-semantics-enforced>true</openstack-vni-semantics-enforced>
<l2gw-stale-vlan-cleanup-delay-secs>900</l2gw-stale-vlan-cleanup-delay-secs>
<l2gw-logical-switch-delay-secs>20</l2gw-logical-switch-delay-secs>
</elanmanager-config>Details
Details
Assignee
Reporter
Labels
Components
Fix versions
Priority
High
It's not clear (yet) what kind of problem (if any) this Exception poses, but the netvirt
clustering CSIT jobs are unstable. The following Exception stood out as something
scary. Potentially, missing flows could be one reason some of the openstack
instances do not have proper connectivity and test cases fail.
2018-04-20T06:25:17,932 | ERROR | opendaylight-cluster-data-notification-dispatcher-94 | DefaultConfigPusher | 374 - org.opendaylight.openflowplugin.applications.of-switch-config-pusher - 0.6.1 | Future (eventually) failed: addFlow org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException: No implementation of RPC AbsoluteSchemaPath{path=[(urn:opendaylight:module:config?revision=2014-10-15)set-config]} available at org.opendaylight.controller.md.sal.dom.broker.impl.RoutedDOMRpcRoutingTableEntry.invokeRpc(RoutedDOMRpcRoutingTableEntry.java:85) [218:org.opendaylight.controller.sal-broker-impl:1.7.1] at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:178) [218:org.opendaylight.controller.sal-broker-impl:1.7.1] at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102) [218:org.opendaylight.controller.sal-broker-impl:1.7.1] at Proxyaec100e4_e760_48b8_b4f7_2f391b651b90.invokeRpc(Unknown Source) [?:?] at Proxyc11c9ab5_3bc0_4a35_92fb_52a7d17ea7ca.invokeRpc(Unknown Source) [?:?] at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:68) [215:org.opendaylight.controller.sal-binding-broker-impl:1.7.1] at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:46) [215:org.opendaylight.controller.sal-binding-broker-impl:1.7.1] at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:165) [215:org.opendaylight.controller.sal-binding-broker-impl:1.7.1] at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:99) [215:org.opendaylight.controller.sal-binding-broker-impl:1.7.1] at com.sun.proxy.$Proxy97.setConfig(Unknown Source) [386:org.opendaylight.openflowplugin.model.flow-service:0.6.1] at org.opendaylight.openflowplugin.openflow.ofswitch.config.DefaultConfigPusher.onDataTreeChanged(DefaultConfigPusher.java:84) [374:org.opendaylight.openflowplugin.applications.of-switch-config-pusher:0.6.1] at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41) [215:org.opendaylight.controller.sal-binding-broker-impl:1.7.1] at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:67) [228:org.opendaylight.controller.sal-distributed-datastore:1.7.1] at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:41) [228:org.opendaylight.controller.sal-distributed-datastore:1.7.1] at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:38) [221:org.opendaylight.controller.sal-clustering-commons:1.7.1] at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.Actor.aroundReceive(Actor.scala:517) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.Actor.aroundReceive$(Actor.scala:515) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.ActorCell.invoke(ActorCell.scala:559) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.run(Mailbox.scala:224) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [42:com.typesafe.akka.actor:2.5.11] 2018-04-20T06:25:17,933 | INFO | ForkJoinPool-1-worker-7 | FlowBasedEgressServicesStateBindHelper | 252 - org.opendaylight.genius.interfacemanager-impl - 0.4.1 | bindServicesOnInterfaceType Egress - WIP 2018-04-20T06:25:17,934 | ERROR | opendaylight-cluster-data-notification-dispatcher-103 | LLDPSpeaker | 373 - org.opendaylight.openflowplugin.applications.lldp-speaker - 0.6.1 | Future (eventually) failed: transmitPacket org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException: Action Local{type=AbsoluteSchemaPath{path=[(urn:opendaylight:packet:service?revision=2013-07-09)transmit-packet]}, contextReference=/(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:275486970454348}]} has no instance matching org.opendaylight.controller.md.sal.binding.impl.LazySerializedContainerNode$WithContextRef@4a0d2b2 at org.opendaylight.controller.blueprint.ext.ActionProviderBean.lambda$registerFallback$0(ActionProviderBean.java:133) ~[195:org.opendaylight.controller.blueprint:0.8.1] at org.opendaylight.controller.md.sal.dom.broker.impl.RoutedDOMRpcRoutingTableEntry.invokeRpc(RoutedDOMRpcRoutingTableEntry.java:71) [218:org.opendaylight.controller.sal-broker-impl:1.7.1] at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:178) [218:org.opendaylight.controller.sal-broker-impl:1.7.1] at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102) [218:org.opendaylight.controller.sal-broker-impl:1.7.1] at Proxyaec100e4_e760_48b8_b4f7_2f391b651b90.invokeRpc(Unknown Source) [?:?] at Proxyc11c9ab5_3bc0_4a35_92fb_52a7d17ea7ca.invokeRpc(Unknown Source) [?:?] at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:68) [215:org.opendaylight.controller.sal-binding-broker-impl:1.7.1] at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:46) [215:org.opendaylight.controller.sal-binding-broker-impl:1.7.1] at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:165) [215:org.opendaylight.controller.sal-binding-broker-impl:1.7.1] at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:99) [215:org.opendaylight.controller.sal-binding-broker-impl:1.7.1] at com.sun.proxy.$Proxy133.transmitPacket(Unknown Source) [386:org.opendaylight.openflowplugin.model.flow-service:0.6.1] at org.opendaylight.openflowplugin.applications.lldpspeaker.LLDPSpeaker.nodeConnectorAdded(LLDPSpeaker.java:190) [373:org.opendaylight.openflowplugin.applications.lldp-speaker:0.6.1] at org.opendaylight.openflowplugin.applications.lldpspeaker.NodeConnectorInventoryEventTranslator.notifyNodeConnectorAppeared(NodeConnectorInventoryEventTranslator.java:179) [373:org.opendaylight.openflowplugin.applications.lldp-speaker:0.6.1] at org.opendaylight.openflowplugin.applications.lldpspeaker.NodeConnectorInventoryEventTranslator.processAddedConnector(NodeConnectorInventoryEventTranslator.java:124) [373:org.opendaylight.openflowplugin.applications.lldp-speaker:0.6.1] at org.opendaylight.openflowplugin.applications.lldpspeaker.NodeConnectorInventoryEventTranslator.onDataTreeChanged(NodeConnectorInventoryEventTranslator.java:102) [373:org.opendaylight.openflowplugin.applications.lldp-speaker:0.6.1] at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41) [215:org.opendaylight.controller.sal-binding-broker-impl:1.7.1] at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:67) [228:org.opendaylight.controller.sal-distributed-datastore:1.7.1] at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:41) [228:org.opendaylight.controller.sal-distributed-datastore:1.7.1] at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:38) [221:org.opendaylight.controller.sal-clustering-commons:1.7.1] at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.Actor.aroundReceive(Actor.scala:517) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.Actor.aroundReceive$(Actor.scala:515) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) [42:com.typesafe.akka.actor:2.5.11] at akka.actor.ActorCell.invoke(ActorCell.scala:559) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.run(Mailbox.scala:224) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [42:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [42:com.typesafe.akka.actor:2.5.11]The above trace was taken from this karaf log from this job. The other two ODL karaf logs are here and here.