flume

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		 Previous revision
flume [2020/11/11 11:21] – created andonovjflume [2020/11/12 11:53] (current) – andonovj
Line 3: Line 3:
  
  
 +=====Management=====
  
 +====Startup====
 +<Code:bash|Startup Flume Server - Terminal won't return to bash>
 +[oracle@edvmr1p0 ~]$ flume-ng agent --conf /opt/flume/conf -f /opt/flume/conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent1
 +Info: Sourcing environment configuration script /opt/flume/conf/flume-env.sh
 +Info: Including Hadoop libraries found via (/opt/hadoop/bin/hadoop) for HDFS access
 +Info: Including HBASE libraries found via (/opt/hbase/bin/hbase) for HBASE access
 ++ exec /usr/java/latest/bin/java -Xms100m -Xmx2000m -Dcom.sun.management.jmxremote -Dflume.root.logger=DEBUG,console -cp '/opt/flume/conf:/opt/flume/lib/*:/opt/hadoop/etc/hadoop:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/hdfs:/opt/hadoop/
 +</Code>
 +This terminal won't exit back to shell. Also you can use teh following command if you want more logging:
 +
 +<Code:bash|Startup Flume v2>
 +[oracle@edvmr1p0 conf]$ flume-ng agent --conf /opt/flume/conf -f /opt/flume/conf/flume.conf -Dflume.root.logger=DEBUG,LOGFILE -n agent1 -Dorg.apache.flume.log.rawdata=true
 +Info: Sourcing environment configuration script /opt/flume/conf/flume-env.sh
 +Info: Including Hadoop libraries found via (/opt/hadoop/bin/hadoop) for HDFS access
 +Info: Including HBASE libraries found via (/opt/hbase/bin/hbase) for HBASE access
 ++ exec /usr/java/latest/bin/java -Xms100m -Xmx2000m -Dcom.sun.management.jmxremote -Dflume.root.logger=DEBUG,LOGFILE -Dorg.apache.flume.log.rawdata=true -cp '/opt/flume/conf:/opt/flume/lib/*:/opt/hadoop/etc/hadoop:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/hdfs:/opt/hadoop/share/hadoop/hdfs/lib/*:/opt/hadoop/share/hadoop/hdfs/*:/opt/hadoop/share/hadoop/yarn/lib/*:/opt/hadoop/share/hadoop/yarn/*:/opt/hadoop/share/hadoop/mapreduce/lib/*:/opt/ha
 +</Code>
 +
 +====Transmit Data====
 +<Code:bash|Transmit Data (e.g. /etc/passwd in that case) - Terminal won't return to bash>
 +[oracle@edvmr1p0 ~]$ flume-ng avro-client --conf conf -H localhost -p 41414 -F /etc/passwd
 +Info: Including Hadoop libraries found via (/opt/hadoop/bin/hadoop) for HDFS access
 +Info: Including HBASE libraries found via (/opt/hbase/bin/hbase) for HBASE access
 ++ exec /usr/java/latest/bin/java -Xmx20m -cp 'conf:/opt/flume/lib/*:/opt/hadoop/etc/hadoop:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/hdfs:/opt/hadoop/share/hadoop/hdfs/lib/*:/
 +</Code>
 +
 +The transmitting of the data will be visible on the previous terminal as follows:
 +<Code:bash|Review Transmitted Data>
 +2020-11-11 11:26:40,702 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 70 75 6C 73 65 3A 78 3A 34 39 37 3A 34 39 33 3A pulse:x:497:493: }
 +2020-11-11 11:26:40,702 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 67 64 6D 3A 78 3A 34 32 3A 34 32 3A 3A 2F 76 61 gdm:x:42:42::/va }
 +2020-11-11 11:26:40,702 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 6F 72 61 63 6C 65 3A 78 3A 35 34 33 32 31 3A 35 oracle:x:54321:5 }
 +2020-11-11 11:26:40,704 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 61 70 61 63 68 65 3A 78 3A 34 38 3A 34 38 3A 41 apache:x:48:48:A }
 +2020-11-11 11:26:40,704 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 61 76 61 68 69 3A 78 3A 37 30 3A 37 30 3A 41 76 avahi:x:70:70:Av }
 +2020-11-11 11:26:40,704 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 61 76 61 68 69 2D 61 75 74 6F 69 70 64 3A 78 3A avahi-autoipd:x: }
 +</Code>
  
 =====Appendix===== =====Appendix=====
  • flume.1605093708.txt.gz
  • Last modified: 2020/11/11 11:21
  • by andonovj