Re: [Cflowd] netflow question - Cisco file format

From: Edwin D. Vinas (edwinv@asti.dost.gov.ph)
Date: Sun Mar 23 2003 - 16:47:09 PST

  • Next message: patricia.kan@HydroOne.com: "RE: [Cflowd] Output Bytes from Arts++"

    ----- Original Message -----
    From: "gab.seun jones.ewulomi" <seun_ewulomi@hotmail.com>
    To: <edwinv@asti.dost.gov.ph>
    Cc: <cflowd@caida.org>
    Sent: Friday, March 21, 2003 6:04 PM
    Subject: Re: [Cflowd] netflow question - Cisco file format

    >
    > Hi Edwin,
    >
    > My apologies for not replying sooner.
    >
    > youre correct that it is quite difficult to install a part-by-part netflow
    > system. I actually got cflowd, arts++, flowscan to work but ran into
    memory
    > issues cflowd was running at 95% cpu contantly. Im actually building
    cflowd
    > on another linux box. It was quite unfortunate that cflowd doesnt give per
    > subnet statistics. I was told flow-tools could do this. I willbe
    > investigating further.
    Yeah, i think flow-tools has many important features that's not in cflowd.

    >
    > I actually checked out the Aguri netflow tool. Excellent tool.
    > Does it actaully read raw netflow exported data/flows?
    > do you have to tell it what file to read?
    I haven't tried using Aguri but i think this tool has different function
    than that of cflowd. Aguri lets you graph the traffic per source & dest IPs.
    In clowd/flowscan you can graph the protocols and services. Maybe we can
    combine the functionalities of these tools.

    >
    > Any pointers will be graetly appreciated.
    >
    > The problem with netflow is the amount of data it generates. What database
    > do use to store aggregated flows e.g. PostgreSQL
    >
    You're right netflow generates big amount of data especially if you are
    exporting un-aggregated data. Some folks says we can apply aggregation in
    the routers so that the data is somewhat smaller. But I never tried applying
    aggregation in the router itself coz im afraid i will not be getting per
    packet granularity. In our case, one day worth of netflow database from one
    router is 500MB+. This is why we always run out of disk space. And that's
    what Im finding out... to properly archive netflow data such that you will
    not sacrifice the details of the database by compressing it. Abilene's
    netflow is one good netflow implementation. They have these so called
    "Weekly Netflow Reports".

    > regards,
    > gab
    >

    best regards,
    edwin

    >
    >
    > >From: "Edwin D. Vinas" <edwinv@asti.dost.gov.ph>
    > >To: "gab.seun jones.ewulomi" <seun_ewulomi@hotmail.com>
    > >CC: <cflowd@caida.org>
    > >Subject: Re: [Cflowd] netflow question - Cisco file format
    > >Date: Fri, 14 Mar 2003 08:45:28 +0800
    > >
    > >hi gab,
    > > >
    > > > Im currently on the verge of installing cflowd and flowscan. your
    > >website
    > > > and netflow implementation has given a breath of encouragement. I
    wanted
    > >to
    > > > give up.
    > > >
    > >Thanks, but don't give up. Its really quite difficult to install a
    > >part-by-part netflow system. I mean cflowd, arts++, flowscan, flowsql,
    > >flow-export configuration, generating summaries etc etc. I think this is
    > >the
    > >price of using all open-source packages instead of commercial netflow
    > >software.
    > >
    > > > 1)Can/Does cflowd/flowscan show per subnet statistics or
    > > > is flow-tools capable of this
    > > >
    > >
    > >I think there are other tools that can do this. Im not sure which one.
    > >However, you can check Aguri netflow tool. It can generate statistics for
    > >each subnet or IPs. I haven't tried generating subnet statistics though.
    > >
    > > > 2)The top summaries how is this generated. what other tools have you
    > > > installed/integarted with netflow to get/generate this data in the
    > >tabular
    > > > format
    > > >
    > >To generate these summaries, I used PHP/Perl to query and summarize the
    top
    > >summaries from the netflow database generated by FlowSQL.
    > >Im also still searching for better methods to incorporate in my netflow
    > >system coz it consumes too much disk space. Also, i will still have to
    > >make an automatic netflow analyzer that could detect spamming, dos
    attacks
    > >and emails a summary report at a regular interval. Right now, Im
    > >just collecting from a main gateway router and a single day database
    > >consumes at least 500MB of my database. This database is the detailed
    > >database which we use for future forensics (i.e., tracing and analyzing
    > >data
    > >sources, protocols, spammers, etc).
    > >
    > >best regards,
    > >edwin
    > >
    > > >
    > > >
    > > >
    > > >
    > > >
    > > >
    > > >
    > > > >From: "Edwin D. Vinas" <edwinv@asti.dost.gov.ph>
    > > > >To: "Vladimir Jirasek" <Vladimir.Jirasek@t-mobile.co.uk>
    > > > >CC: <cflowd@caida.org>
    > > > >Subject: Re: [Cflowd] netflow question - Cisco file format
    > > > >Date: Thu, 13 Mar 2003 08:21:41 +0800
    > > > >
    > > > >MessageHi,
    > > > >
    > > > >If you want collect netflow from routers, it is possible to use
    Cflowd.
    > > > >Cflowd has two components -- cflowdmux & cfdcollect. When cflowd is
    > > > >running, it will collect raw flow files version 5 format from
    > > > >flow-exporters and saves the raw flow files in arts++ format. In our
    > >case
    > > > >we are using Cflow to analyze these raw flow files. To graph the data
    > >you
    > > > >can use FlowScan and to database it we used a custom program called
    > >FlowSQL
    > > > >which stores the granular flow fields in a Postgresql database. This
    is
    > >an
    > > > >example implemenation: http://noc.asti.dost.gov.ph/netflow/index.php
    > >Docs:
    > > > >http://netmeas.asti.dost.gov.ph/docus/netflow/Netflow.pdf
    > > > >
    > > > >HTH :-)
    > > > >
    > > > >best regards,
    > > > >--edwin
    > > > >
    > > > >-----------------------------------------------------------------
    > > > >If Americans have atomic bombs & the Internet...
    > > > >Filipinos are very far behind to catch up in any field.
    > > > >-Edwin D. Viņas
    > > > >edwinv@asti.dost.gov.ph
    > > > >http://www.geocities.com/edwin_vinas
    > > > >Science Research Specialist I
    > > > >PREGINET Project
    > > > >Advanced Science and Technology Institute
    > > > >UP Technopark Complex, CP Garcia Ave, Diliman,
    > > > >Quezon City Philippines
    > > > >-----------------------------------------------------------------
    > > > >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    > > > >This communication is intended only for the person or entity to which
    > >it
    > >is
    > > > >addressed and may contain confidential and/or privileged material.
    If
    > >you
    > > > >are not the intended recipient, please note that any review,
    > > > >retransmission,
    > > > >dissemination, copying or other use of, or taking of any action in
    > >reliance
    > > > >upon, this information by you or by persons or entities other than
    the
    > > > >intended recipient is prohibited.
    > > > >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    > > > >
    > > > > ----- Original Message -----
    > > > > From: Vladimir Jirasek
    > > > > To: 'cflowd@caida.org'
    > > > > Sent: Wednesday, March 12, 2003 10:04 PM
    > > > > Subject: [Cflowd] netflow question - Cisco file format
    > > > >
    > > > >
    > > > > Hi,
    > > > >
    > > > > I have developed the tool that can read text files from Cisco
    > >Netflow
    > > > >collector via ftp and analyse it down to application flow level. Now
    I
    > >want
    > > > >to setup something similar but using Cflowd. I would like to use only
    > > > >collector function and get raw data prefferably in Cisco format.
    > > > > Is this achievable?
    > > > >
    > > > > Many thanks
    > > > >
    > > > > Vladimir Jirasek
    > > > > Mobile: +447956542287
    > > > > Fixed line: +442082142813
    > > > > International Workgroup Corporate network (EU153)
    > > > > T-Mobile International
    > > > > Imperial place, Borehamwood, WD61EA
    > > > > United Kingdom
    > > > >
    > > > >
    > > > >
    > > > >
    > > > >
    > > > > NOTICE AND DISCLAIMER:
    > > > >
    > > > > This email (including attachments) is confidential. If you have
    > >received
    > > > >this email in error please notify the sender immediately and delete
    > >this
    > > > >email from your system without copying or disseminating it or placing
    > >any
    > > > >reliance upon its contents. We cannot accept liability for any
    breaches
    > >of
    > > > >confidence arising through use of email. Any opinions expressed in
    this
    > > > >email (including attachments) are those of the author and do not
    > > > >necessarily reflect our opinions. We will not accept responsibility
    for
    > >any
    > > > >commitments made by our employees outside the scope of our business.
    We
    > >do
    > > > >not warrant the accuracy or completeness of such information.
    > > > >
    > > > >
    > > > >
    > > >
    > > >
    > > > _________________________________________________________________
    > > > Worried what your kids see online? Protect them better with MSN 8
    > > >
    >
    >http://join.msn.com/?page=features/parental&pgmarket=en-gb&XAPID=186&DI=105
    9
    > > >
    > >
    >
    >
    > _________________________________________________________________
    > It's fast, it's easy and it's free. Get MSN Messenger today!
    > http://messenger.msn.co.uk
    >

    _______________________________________________
    Cflowd mailing list
    Cflowd@caida.org
    http://login.caida.org/mailman/listinfo/cflowd



    This archive was generated by hypermail 2.1.4 : Sun Mar 23 2003 - 16:57:54 PST