1
0
Fork 0
mirror of https://github.com/moby/moby.git synced 2022-11-09 12:21:53 -05:00
Commit graph

19 commits

Author SHA1 Message Date
Justin Menga
4374f16667 Append new line to multiline events
Signed-off-by: Justin Menga <justin.menga@gmail.com>
2017-05-13 17:54:04 +12:00
Justin Menga
8ef9c19ecd Code review changes
Signed-off-by: Justin Menga <justin.menga@gmail.com>
2017-05-13 17:54:04 +12:00
Tibor Vass
f775005a17 awslogs: use github.com/pkg/errors to wrap an error
Signed-off-by: Tibor Vass <tibor@docker.com>
2017-05-13 17:54:04 +12:00
Justin Menga
ab74038df9 Add awslogs multiline support
Signed-off-by: Justin Menga <justin.menga@gmail.com>
2017-05-13 17:54:04 +12:00
Brian Goff
3f4fccb65f Use sync.Pool for logger Messages
This reduces allocs and bytes used per log entry significantly as well
as some improvement to time per log operation.

Each log driver, however, must put messages back in the pool once they
are finished with the message.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2017-02-01 13:52:37 -05:00
Brian Goff
054abff3b6 Implement optional ring buffer for container logs
This allows the user to set a logging mode to "blocking" (default), or
"non-blocking", which uses the ring buffer as a proxy to the real log
driver.

This allows a container to never be blocked on stdio at the cost of
dropping log messages.

Introduces 2 new log-opts that works for all drivers, `log-mode` and
`log-size`. `log-mode` takes a  value of "blocking", or "non-blocking"
I chose not to implement this as a bool since it is difficult to
determine if the mode was set to false vs just not set... especially
difficult when merging the default daemon config with the container config.
`log-size` takes a size string, e.g. `2MB`, which sets the max size
of the ring buffer. When the max size is reached, it will start
dropping log messages.

```
BenchmarkRingLoggerThroughputNoReceiver-8           	2000000000	        36.2 ns/op	 856.35 MB/s	       0 B/op	       0 allocs/op
BenchmarkRingLoggerThroughputWithReceiverDelay0-8   	300000000	       156 ns/op	 198.48 MB/s	      32 B/op	       0 allocs/op
BenchmarkRingLoggerThroughputConsumeDelay1-8        	2000000000	        36.1 ns/op	 857.80 MB/s	       0 B/op	       0 allocs/op
BenchmarkRingLoggerThroughputConsumeDelay10-8       	1000000000	        36.2 ns/op	 856.53 MB/s	       0 B/op	       0 allocs/op
BenchmarkRingLoggerThroughputConsumeDelay50-8       	2000000000	        34.7 ns/op	 894.65 MB/s	       0 B/op	       0 allocs/op
BenchmarkRingLoggerThroughputConsumeDelay100-8      	2000000000	        35.1 ns/op	 883.91 MB/s	       0 B/op	       0 allocs/op
BenchmarkRingLoggerThroughputConsumeDelay300-8      	1000000000	        35.9 ns/op	 863.90 MB/s	       0 B/op	       0 allocs/op
BenchmarkRingLoggerThroughputConsumeDelay500-8      	2000000000	        35.8 ns/op	 866.88 MB/s	       0 B/op	       0 allocs/op
```

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2017-02-01 13:52:37 -05:00
Yong Tang
864b0c2e47 Support of CreateLogGroup for awslogs
This fix tries to address the issue raised in 29344 where it was
not possible to create log group for awslogs (CloudWatch) on-demand.
Log group has to be created explicitly before container is running.

This behavior is inconsistent with AWS logs agent where log groups
are always created as needed.

There were several concerns previously (See comments in 19617 and 29344):
1. There is a limit of 500 log groups/account/region so resource might
be exhausted if there is any typo or incorrect region.
2. Logs are generated for every container so CreateLogGroup (or equally,
DescribeLogGroups) might be called every time, which is redundant and
potentially surprising.
3. CreateLogStream and CreateLogGroup have different IAM policies.

This fix addresses the issue by add `--log-opt awslogs-create-group`
which by default is `false`. It requires user to explicitly request
that log groups be created as needed.

Related unit test has been updated. And tests have also been done
manually in AWS.

This fix fixes 29334.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
2017-01-27 11:15:17 -08:00
Yanqiang Miao
17ec911da7 Rename 'context' to 'loginfo' in the logger module
Signed-off-by: Yanqiang Miao <miao.yanqiang@zte.com.cn>

gofmt

Signed-off-by: Yanqiang Miao <miao.yanqiang@zte.com.cn>

update

Signed-off-by: Yanqiang Miao <miao.yanqiang@zte.com.cn>

remove 'api/types/container/config.go' from this PR

Signed-off-by: Yanqiang Miao <miao.yanqiang@zte.com.cn>

change 'LogInfo' to 'Info'

Signed-off-by: Yanqiang Miao <miao.yanqiang@zte.com.cn>

update

Signed-off-by: Yanqiang Miao <miao.yanqiang@zte.com.cn>
2016-12-29 19:13:44 +08:00
French Ben
3661510f7f Updated AWS logstream to understand tags
Signed-off-by: French Ben <frenchben@docker.com>
2016-11-03 13:49:40 -07:00
Samuel Karp
443f251cf5 awslogs: Record log line insert order for sorting
Fixes https://github.com/docker/docker/issues/24775

Signed-off-by: Samuel Karp <skarp@amazon.com>
2016-08-03 10:04:20 -07:00
Nalin Dahyabhai
513ec73831 Improve logging of long log lines
This change updates how we handle long lines of output from the
container.  The previous logic used a bufio reader to read entire lines
of output from the container through an intermediate BytesPipe, and that
allowed the container to cause dockerd to consume an unconstrained
amount of memory as it attempted to collect a whole line of output, by
outputting data without newlines.

To avoid that, we replace the bufio reader with our own buffering scheme
that handles log lines up to 16k in length, breaking up anything longer
than that into multiple chunks.  If we can dispense with noting this
detail properly at the end of output, we can switch from using
ReadBytes() to using ReadLine() instead.  We add a field ("Partial") to
the log message structure to flag when we pass data to the log driver
that did not end with a newline.

The Line member of Message structures that we pass to log drivers is now
a slice into data which can be overwritten between calls to the log
driver's Log() method, so drivers which batch up Messages before
processing them need to take additional care: we add a function
(logger.CopyMessage()) that can be used to create a deep copy of a
Message structure, and modify the awslogs driver to use it.

We update the jsonfile log driver to append a "\n" to the data that it
logs to disk only when the Partial flag is false (it previously did so
unconditionally), to make its "logs" output correctly reproduce the data
as we received it.

Likewise, we modify the journald log driver to add a data field with
value CONTAINER_PARTIAL_MESSAGE=true to entries when the Partial flag is
true, and update its "logs" reader to refrain from appending a "\n" to
the data that it retrieves if it does not see this field/value pair (it
also previously did this unconditionally).

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2016-06-14 14:11:47 -04:00
Yong Tang
46ea8ff75d Fix build errors caused by update aws-sdk-go to v1.1.30
This fix tries to fix build errors caused by updating
aws-sdk-go to v1.1.30.

This fix fixes #22961.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
2016-05-28 18:49:17 -07:00
Vincent Demeester
8054a30387 dockerversion placeholder for library import
- Add a *version* file placeholder.
- Update autogen and builds to use it and an autogen build flag

Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2015-11-09 19:32:46 +01:00
John Howard
2716e3964d Windows build broken on master (awslogs)
Signed-off-by: John Howard <jhoward@microsoft.com>
2015-10-30 10:02:32 -07:00
Samuel Karp
8a6dfb26f3 [awslogs] Auto-detect region on EC2 instances
Signed-off-by: Samuel Karp <skarp@amazon.com>
2015-10-27 23:47:28 +00:00
Samuel Karp
480c9c0178 [awslogs] Set User-Agent for Amazon CloudWatch Logs
Signed-off-by: Samuel Karp <skarp@amazon.com>
2015-10-27 23:47:19 +00:00
Samuel Karp
70083e6fc3 Update vendored aws-sdk-go
Signed-off-by: Samuel Karp <skarp@amazon.com>
2015-10-27 23:43:51 +00:00
Alexander Morozov
32964def83 awslogs: fix logrus import
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
2015-09-11 11:05:06 -07:00
Samuel Karp
3effe484e6 Add awslogs driver for Amazon CloudWatch Logs
Signed-off-by: Samuel Karp <skarp@amazon.com>
2015-09-09 13:52:40 -07:00