Due to a slice management logic error the bulk sync for loop can go on
indefinitely and eventually leading to an OOM error. Fixed the logic so
that an infinite loop never occurs. Also changed the bulk sync wait
timeout to use a timer rather than use time.After as time.After is known
to consume a lot of memory when called in a tight loop.
Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
When a node goes away purge all the network attachments from the node
and make sure we don't attempt bulk syncing to that node once removed.
Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
When a node goes away we purge all the table entries that we learned
from that node but we don't notify the watchers about it. Made sure we
notify the watchers when this happens.
Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
When broadcasting table event, make sure the broadcast queue is
valid. The network may have been removed while in the process of sending
the broadcast.
Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
The wait in bulkSyncNode was meant for bulkSync initiator. Not for
responder. Fix the incorrect code which was also waiting unnecessarily
on response which it will never get and will eventually time out.
Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
Convert all networkdb core message types from go message types to
protobuf message types. This faciliates future modification of the
message structure without breaking backward compatibility.
Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
libnetwork agent mode is a mode where libnetwork can act as a local
agent for network and discovery plumbing alone while the state
management is done elsewhere. This completes the support for making
libnetwork and its associated drivers to be completely independent of a
k/v store(if needed) and work purely based on the state information
passed along by some some external controller or manager. This does not
mean that libnetwork support for decentralized state management via a
k/v store is removed.
Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
When a node joins a network it sends out a gossip event before it
updates it's own in-memory state. This can create a race where the node
gets the event back from a remote node before we update in-memory state
and we treat that as latest state. To avoid this race, always generate
the gossip after updating local state.
Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
Network DB is a network scoped gossip database built
on top of hashicorp/memberlist providing an eventually
consistent state store.
It limits the scope of the gossip and periodic bulk syncing
for table entries to only the nodes which participate in the
network to which the gossip belongs. This designs make the
gossip layer scale better and only consumes resources for the
network state that the node participates in.
Since the complete state for a network is maintained by all nodes
participating in the network, all nodes will eventually converge
to the same state.
NetworkDB also provides facilities for the users of the package to
watch on any table (or all tables) and get notified if there are
state changes of interest that happened anywhere in the cluster when
that state change eventually finds it's way to the watcher's node.
Signed-off-by: Jana Radhakrishnan <mrjana@docker.com>
