Chris Taylor
2018-08-29 19:39:03 UTC
We recently moved our Apache web servers to vmware from ganetti and
basically cloned everything.
However, we're getting weird waits from the Apache server. Here's what an
strace looks like FROM the database server:
The read(14 is the network connection to the Apache server.
The Event in the database is "SQL*Net MORE Data From Client"
A big, big object is built on the apache side and passed over the network
to the database. This object is broken up into chunks.
When on Ganetti we didn't see these network waits... each chunk would come
across pretty fast. But now, there's a significant wait for each chunk.
I'm looking to see if anyone had "tune" Apache/VMWare configurations
talking across SQLNet to Oracle and if so, what did you tweak? We've
changed the network card from E1000 to VMXnet (something, something) and
its bridged, not NAT. I think that improved things somewhat but still lots
waits between each READ from CLIENT connection.
So each read(14 is reading the open network connection to the Apache
server. Used lsof to find what "14" was here.
The web/app server has built a very large object that we store in a LOB
The object is base64 encoded before it is passed to Oracle
120.304233 read(14, "stripped", 553) = 553
120.319649 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={0, 215967},
ru_stime={0, 94985}, ru_maxrss=27116, ru_ixrss=0, ru_idrss=0, ru_isrss=0,
ru_minflt=9792, ru_majflt=0, ru_nswap=0, ru_inblock=896, ru_oublock=0,
ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=285, ru_nivcsw=17}) = 0
0.000619 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={0, 215967},
ru_stime={0, 94985}, ru_maxrss=27116, ru_ixrss=0, ru_idrss=0, ru_isrss=0,
ru_minflt=9792, ru_majflt=0, ru_nswap=0, ru_inblock=896, ru_oublock=0,
ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=287, ru_nivcsw=17}) = 0
0.000038 read(14, "stripped", 8015) = 2920
120.319281 read(14, "stripped", 5095) = 2920
120.324230 read(14, "stripped", 2175) = 2175
120.316067 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={0, 215967},
ru_stime={0, 95985}, ru_maxrss=27116, ru_ixrss=0, ru_idrss=0, ru_isrss=0,
ru_minflt=9792, ru_majflt=0, ru_nswap=0, ru_inblock=896, ru_oublock=0,
ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=312, ru_nivcsw=17}) = 0
0.000274 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={0, 215967},
ru_stime={0, 95985}, ru_maxrss=27116, ru_ixrss=0, ru_idrss=0, ru_isrss=0,
ru_minflt=9792, ru_majflt=0, ru_nswap=0, ru_inblock=896, ru_oublock=0,
ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=314, ru_nivcsw=17}) = 0
basically cloned everything.
However, we're getting weird waits from the Apache server. Here's what an
strace looks like FROM the database server:
The read(14 is the network connection to the Apache server.
The Event in the database is "SQL*Net MORE Data From Client"
A big, big object is built on the apache side and passed over the network
to the database. This object is broken up into chunks.
When on Ganetti we didn't see these network waits... each chunk would come
across pretty fast. But now, there's a significant wait for each chunk.
I'm looking to see if anyone had "tune" Apache/VMWare configurations
talking across SQLNet to Oracle and if so, what did you tweak? We've
changed the network card from E1000 to VMXnet (something, something) and
its bridged, not NAT. I think that improved things somewhat but still lots
waits between each READ from CLIENT connection.
So each read(14 is reading the open network connection to the Apache
server. Used lsof to find what "14" was here.
The web/app server has built a very large object that we store in a LOB
The object is base64 encoded before it is passed to Oracle
120.304233 read(14, "stripped", 553) = 553
120.319649 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={0, 215967},
ru_stime={0, 94985}, ru_maxrss=27116, ru_ixrss=0, ru_idrss=0, ru_isrss=0,
ru_minflt=9792, ru_majflt=0, ru_nswap=0, ru_inblock=896, ru_oublock=0,
ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=285, ru_nivcsw=17}) = 0
0.000619 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={0, 215967},
ru_stime={0, 94985}, ru_maxrss=27116, ru_ixrss=0, ru_idrss=0, ru_isrss=0,
ru_minflt=9792, ru_majflt=0, ru_nswap=0, ru_inblock=896, ru_oublock=0,
ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=287, ru_nivcsw=17}) = 0
0.000038 read(14, "stripped", 8015) = 2920
120.319281 read(14, "stripped", 5095) = 2920
120.324230 read(14, "stripped", 2175) = 2175
120.316067 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={0, 215967},
ru_stime={0, 95985}, ru_maxrss=27116, ru_ixrss=0, ru_idrss=0, ru_isrss=0,
ru_minflt=9792, ru_majflt=0, ru_nswap=0, ru_inblock=896, ru_oublock=0,
ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=312, ru_nivcsw=17}) = 0
0.000274 getrusage(0x1 /* RUSAGE_??? */, {ru_utime={0, 215967},
ru_stime={0, 95985}, ru_maxrss=27116, ru_ixrss=0, ru_idrss=0, ru_isrss=0,
ru_minflt=9792, ru_majflt=0, ru_nswap=0, ru_inblock=896, ru_oublock=0,
ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=314, ru_nivcsw=17}) = 0