Key ingredients to achieve effective 1/O The complexity of asynchronous I/O Some consequences o involves multi-threaded multi-process approach o with a background task for the actual transfer. o involves out of order transfers o no garantee on the order in which transfers will arrive to destination especially in case of failures o involves queue management on the caller side to not send too much in parallel to adapt to network efficiency Libraries Do not try to do all that by hand Use a library that does(most of)it for you latancy asyne mig 11/57 S.Ponce-CERN
Key ingredients to achieve effective I/O 11 / 57 S. Ponce - CERN async IOopt struct caching c/c latency async msgQ The complexity of asynchronous I/O Some consequences involves multi-threaded / multi-process approach with a background task for the actual transfer. involves out of order transfers no garantee on the order in which transfers will arrive to destination especially in case of failures involves queue management on the caller side to not send too much in parallel to adapt to network efficiency Libraries Do not try to do all that by hand Use a library that does (most of) it for you
Key ingredients to achieve effective 1/O c/o POSIX aynchronous I/O Old interface based on signals struct aiocb int aio_fildes; /*fd*/ off_t aio_offset; /offset * volatile void *aio_buf; /buffer * size_t aio_nbytes; /length * int aio_reqprio; /priority 1* struct sigevent aio_sigevent; /cb method int aio_lio_opcode;/*operation 米入 } int aio_read (struct aiocb *aiocbp); int aio_write (struct aiocb *aiocbp) The application can elect to be notified of completion of the I/O operation in a variety of ways:by delivery of a signal,by instantiation of a thread,or no notification at all. atancy async mi名 12/57 S.Ponce-CERN
Key ingredients to achieve effective I/O 12 / 57 S. Ponce - CERN async IOopt struct caching c/c latency async msgQ POSIX aynchronous I/O Old interface based on signals struct aiocb { int aio_fildes; /* fd */ off_t aio_offset; /* offset */ volatile void *aio_buf; /* buffer */ size_t aio_nbytes; /* length */ int aio_reqprio; /* priority */ struct sigevent aio_sigevent; /* cb method */ int aio_lio_opcode; /* operation */ }; int aio_read (struct aiocb *aiocbp); int aio_write (struct aiocb *aiocbp) The application can elect to be notified of completion of the I/O operation in a variety of ways: by delivery of a signal, by instantiation of a thread, or no notification at all
Key ingredients to achieve effective I/O 4 asyne IOop时nt caching c/c POSIX aynchronous I/O Internal tweaks o The proc file system contains two virtual files that can be tuned for asynchronous I/O performance: The/proc/sys/fs/aio-nr file provides the current number of system-wide asynchronous l/O requests. The/proc/sys/fs/aio-max-nr file is the maximum number of allowable concurrent requests.The maximum is commonly 64KB,which is adequate for most applications. latancy asyne mig 13/57 S.Ponce-CERN
Key ingredients to achieve effective I/O 13 / 57 S. Ponce - CERN async IOopt struct caching c/c latency async msgQ POSIX aynchronous I/O Internal tweaks The proc file system contains two virtual files that can be tuned for asynchronous I/O performance: The /proc/sys/fs/aio-nr file provides the current number of system-wide asynchronous I/O requests. The /proc/sys/fs/aio-max-nr file is the maximum number of allowable concurrent requests. The maximum is commonly 64KB, which is adequate for most applications
Key ingredients to achieve effective 1/O ay0op时caching c Ceph aynchronous 1/O More modern interface with callbacks typedef void (*rados_callback_t) (rados_completion_t cb,void *arg); int rados_aio_create_completion (void *cb_arg,rados_callback_t cb_complete, rados_callback_t cb_safe, rados_completion_t *pc); int rados_aio_write (rados_ioctx_t io,const char oid, rados_completion_t completion, const char buf,size_t len,uint64_t off); int rados_aio_wait_for_safe(rados_completion_t c); Handling of threading is done for you latency asyne mig 14/57 S.Ponce-CERN
Key ingredients to achieve effective I/O 14 / 57 S. Ponce - CERN async IOopt struct caching c/c latency async msgQ Ceph aynchronous I/O More modern interface with callbacks typedef void (*rados_callback_t) (rados_completion_t cb, void *arg); int rados_aio_create_completion (void *cb_arg, rados_callback_t cb_complete, rados_callback_t cb_safe, rados_completion_t *pc); int rados_aio_write (rados_ioctx_t io, const char * oid, rados_completion_t completion, const char * buf, size_t len, uint64_t off); int rados_aio_wait_for_safe(rados_completion_t c); Handling of threading is done for you
Key ingredients to achieve effective 1/O caching c/c Ceph aynchronous l/O example code void commit_cb(rados_completion_t comp,void *arg){ struct timeval *t =(struct timeval *arg; gettimeofday(&t,NULL); int main(){ ..·/connect to ceph struct timeval commit_end; rados_completion_t comp; rados_aio_create_completion ((void*)&commit_end,NULL,commit_cb,&comp); rados_aio_append(ioctx,obj_name,comp,data,len); rados_aio_flush(io);/wait for call back /print commit_end latancy async m 15/57 S.Ponce-CERN
Key ingredients to achieve effective I/O 15 / 57 S. Ponce - CERN async IOopt struct caching c/c latency async msgQ Ceph aynchronous I/O example code void commit_cb(rados_completion_t comp, void *arg) { struct timeval *t = (struct timeval *) arg; gettimeofday(&t, NULL); } int main() { ... // connect to ceph struct timeval commit_end; rados_completion_t comp; rados_aio_create_completion ((void*) &commit_end, NULL, commit_cb, &comp); rados_aio_append(ioctx, obj_name, comp, data, len); rados_aio_flush(io); // wait for call back // print commit_end }