Re: cflowd memory usage

From: Blake Caldwell (blake@ucar.edu)
Date: Sun Feb 11 2001 - 11:59:37 PST

  • Next message: Mark Fullmer: "Re: cflowd memory usage"

    Hendrik,

    Yes, you are right. That turned out to be the source of the memory leak. My
    modified flowwatch didn't have the same problem because it doesn't have the
    code to create tabular aggregation for cfdcollect. However, my flowwatch
    didn't turn out to be as reliable as I thought. It worked great for 36 hours,
    the it seg faulted, and would not restart without seg faulting. I'm not sure
    what the nature of this problem is, but my best bet would probably to start
    modifying cflowd, as it has more built in functions and doesn't seg fault.

    I probably won't make my version of flowwatch into a patch, unless someone else
    wants to work on it. Right now I am in the process of cleaning up cflowd so
    that it does everything flowscan needs, but no more.
    I believe this can cut the cpu usage by a lot, and use less memory. This is
    important to me, because flowscan needs all the cpu juice it can get! Maybe it
    would be better to combine cflowd and cflowdmux into one program to eliminate
    the intermediate shared memory step. I am relatively inexperienced with c++
    programming, so I would appreciate any help anyone can offer. Has anyone done
    something like this before?

    Thanks,
    Blake

    Hendrik Visage wrote:

    > On Sat, Feb 10, 2001 at 04:35:35PM -0700, Blake Caldwell wrote:
    > > I just found a way to get around all of this. I hacked the flowwatch
    > > program that connects to cflowdmux to write the raw flows to disk. Now I
    > > don't even need to use cflowd! Its much faster and probably more
    > > reliable. I would suggest doing this to anyone who is using Flowscan.
    > > Flowwatch only uses a couple megs of memory compared to 650 Mb with
    > > cflowd. If anyone is interested, let me know.
    >
    > Pathes appreciated, however, had you configured ANY aggregation,
    > ie. net/as/port/etc. ??? Looking at the code, I would say
    > those are the reasons why it's growing in size. If you don't
    > use that, then don't configure it, as it mostly only gets cleaned up
    > after a cfdcollect collected those data and wrote it to disk.
    >
    > Greetz
    > Hendrik
    >
    > >
    > > -Blake
    > >
    > >
    > > Andrew Fort wrote:
    > >
    > > > Oops, sorry I should've posted this to the list, please send any replies
    > > > there
    > > >
    > > > > From: Andrew Fort
    > > > >
    > > > > Blade Caldwell wrote,
    > > > >
    > > > > > Sometimes it will grow
    > > > > > up to 650 Mb in a single day. Because of this I am forced to
    > > > > > restart Cflowd 3
    > > > > > times a day. I know this is more data than cflowd was
    > > > >
    > > > > sounds like a leak; what versions of cflowd and arts++ are in
    > > > > use, on what platform?
    > > > >
    > > > > I had a similar problem, caused by me having cfdcollect and
    > > > > cflowdmux running but NOT cflowd (my collector reads from
    > > > > cflowdmux's shared memory cells). Killing cflowdmux stopped the leak.
    > > > >
    > > > > -afort
    > > > >
    > >
    > > --
    > > cflowd mailing list
    > > cflowd@caida.org
    >
    > --
    > ------------------------
    > Hendrik Visage
    > hvisage@envisage.co.za

    --
    cflowd mailing list
    cflowd@caida.org
    



    This archive was generated by hypermail 2b29 : Sun Feb 11 2001 - 12:13:39 PST