Tuning and development with SIP Servlets on Mobicents

Editor's Notes

1. At first I’m not very fluent in English. So I will look at my notes during the presentation. Please bear with me and feel free to interrupt me and ask any questions. All right, first of all, thanks for your coming this session. Today, I’d like to talk about tuning and development with sip servlet on mobicents. This is my first speech in English. And I participate in JUDCon for the first time too. Please be easy on me Let’s get started.
2. This is agenda. First introduction Second I describe about SIP Application behavior and mobicents sip servlet container. Third How to tune the JavaVM for SIP Servlet Last How to develop SIP Servlet Application with frameworks
3. Introduction I’m mobicents fellow. All you know, mobicents is the most popular OSS community for Telecom. I’ve joined to core team of mobicents since this March. And I’m contributing SSF to mobicents. SSF is for developing SIP Servlet Application with Spring Framework. My company, OKI has developed SIP Servlet and container since 2003 (two thousands and three) (After this, Introduce Jean)
4. Next, The brief description of my company All you would not know OKI, but you may know OKI data. OKI is parent company of OKI data.:-0 Founded, January 1881 (eighteen, eighty one) Major operation is as you can see it.
5. My current works are: Support Japan carriers They are evaluating the oss platform. They are usually very conservative and will be very sensitive and careful for managing and handling error and abnormal cases. But they are very important for platforms. We are analyzing and evaluating their requirements, and how can I feedback them to mobicents. Customize mobicents sip servlet for proprietary. Our customer requires Do not include internal IP address in SIP Header fields. Do not over 256 characters per line in SIP Header. Etceteras Developing customized sip load balancer for Japanese carrier.
6. Let’s get down to business. #Return to the subject. Start to talk about Sip Servlet behavior
7. 1 st , As you know. Some SIP messages will be sent for creating one session. INVITE, 180./INVITE, 200/INVITE, ACK, BYE, 200/BYE and etc…
8. This is most basic b2bua sequence. 細かい質問があったら、 Jean に任せる。 At least, sip specification creates 2 dialogs, 6 transactions, 12 messages for one session. If UDP retransmissions occurred, more messages will be sent and create more objects and discard them
9. In addition, many objects are created.
10. For example, on mobicents sip servlets, This table is instance counts after 10,000 calls are established. These objects are top counts of remained objects related to jain-sip. Almost these objects are SIP header. There are short life time objects, or related to SipSession or SipApplicationSession Related objects will be left on JVM until SipSessions and SipApplicationSession are terminated. Heap will be filled with like these objects.
11. As MSS own behavior, many threads run on MSS. For transport, timer, cluster, http, j2ee container… and so on For Jain-sip and sip-servlets stack, about 100 threads run by default settings. But you can configure and reduce them.
12. SIP with UDP transport has a common problem due to retransmission. In normal case, UDP transport will be used. when sip response sent late over 500ms, retransmission of sip message will coccur. And If jvm was stopped long time by gc, Call failure will be occurred and error handling may not work properly. Or SIP has many timers, and they could not work properly. Retry with http is relatively easier than sip You would have to start over from scratch to initiate SIP session.
13. From these behaviors, the results can be derived as follows. MSS will discard many objects in one session. When retransmissions of UDP messages occurred, Many threads runs at the same time CPU usage go up New GC will be missed Full GC will run If many retransmissions occurred, more objects will discard. GC will be missed more… And fall in a vicious cycle.
14. For these reasons, I will introduce How to tune the JVM to avoid these problems.
15. My plan is Reduce retransmissions of UDP Response time less than 200ms Measure the performance 400call/sec(1440,000BHCA Busy Hour Call Attempts) I want to use multiple cpu effectively Most of server-machine have multiple CPU, and I want to use multiple CPU well with the concurrent GC.
16. This is test sequence that is basic b2bua sequence I described before. Wait 10 seconds after ringing response sent Wait 30 seconds after ACK request sent
17. These are recommended Java VM Options. These options are set, you will reduce the “Stop the World” by GC. UsetTLAB is to use thread-local object allocation These options enable the concurrent mark sweep GC
18. These are other recommended options for mss. When These options is enabled, mobicents sip servlet will work more stable.
19. These are other recommended options These options are for optimization. SIP message parser will create many string objects. I thought these options will be useful for mss.
20. I tried four patterns. And I measured GC, CPU usage, failed call, succeeded calls, retransmission, and response time.
21. First These are almost default gc options for mss Heap memory is set to 6 giga bytes Permanent space size is 256 mega bytes
22. Let see charts. CPU usage has many peaks, but average is 30% to 40% There are over thousands Retransmissions and failed calls “ Stop the world” is occurred. Many responses took more than 200ms.
23. Next, Add the recommended VM options. They enable concurrent gc.
24. These charts are CPU usage is increased, seems to be used more efficiently . Failed call is 0. CPU pause time less than 100ms. Time-consuming response is decreased! Almost happy to me. But many retransmission is still remained.
25. Next I added the CMS tuning options. These options is enabled customized the duty cycle. 100% of time is used for concurrent collector. You will be able to use multiple CPU power more efficiently for GC.
26. These are charts of result. Retransmission is decreased to about 10%. GC pause time is less than 100ms and heap utilization is decreased too. In addition, time-consuming response is decreased. These results is very satisfactory.
27. Next I tried to add more options . These options are for optimization. For more information about these options, please see the Java VM documentation.
28. The result are not any better the previous chart. CPU usage and memory usage is slightly decreased. But retransmissions and time-consuming response is increased.
29. Conclusion Recommended options are useful. In addition, Options to tune the duty cycle are very useful. As mentioned before, multiple CPU can be used effectively. In addition, If you set optimization options instead of duty cycle options, You can see a similar effect. These results may also vary depending on machine spec However, it should be able to get the same effect by tuning the recently server machine Moreover, it is also effective to change the parameter of mss-sip-stack.properties to decrease retransmissions. These retransmissions are caused by udp packet errors. MSS is executing with one thread and one port for udp package transmission. Mobicents will improve this issu in the near future with NIO. まず、最初の VM オプションはかなり有効であることがおわかりいただけたと思います。 さらに duty cycle のチューニングをすることで、大きく性能があがったことがおわかりいただけたでしょう。 また、追加のオプションは少しは役に立ちましたが、 duty cycle のチューニングをしていなかった場合、同程度の性能向上を 行えたこともお伝えしておきます。 今回は VM のオプションを設定することでチューニングを施しました。 しあし、 MSS のオプションによっても性能を向上させることはできます。 再送は UDP の SC_RCVBUF の設定によっても減らすことが可能です。 再送が発生している時、 netstat で取得した情報から、 UDP のパケットエラーが発生していることに気づきました。 SC_RCVBUF の値を倍の値を 131072 にすることで、パケットエラーを減らすことができました。 しかし、０にすることはできませんでした。 この問題は、ある程度のマシンスペックがないと発生しないことが我々の試験でわかっています。 低スペックのマシンでは、それ以前に処理が間に合わずにエラーが発生するためです。 このため、この問題を解決するための方法について考察したことがあります。 MSS は UDP のパケット通信をひとつのポートおよびひとつのスレッドで処理しています。 Rport に対応させ、 UAC として動作するときにデフォルトの受信ポートと違うポートを使用してメッセージを送受信することで、 さらなる性能向上が見込めます。 実際に、 NIO を使用し、 UAC として動作する際にポートを開くようにトランスポートを改造してみたところ、パケットエラーを０にすることに成功しました。 ただ、試験用のためコミュニティに寄贈できるレベルのものを作成することができませんでした。 この提案はすでに行っており、近い将来に MSS で実装されるものと考えています。
30. Let me show you developing sip servlet with frameworks.
31. Mobicents has 2 sip servlet frameworks. First, My project, ssf This framework based on spring framework. Second CTF This framework based on cdi that is using standard api. These frameworks have same concept Simplify SipServlets development. The concept of sip servlet, web developer is also being able to facilitate development of sip. But all you know, sip servlet like the early days http servlet development, you have low-level api for sip development, And development is not easy. We want to accelerate application development.
32. First, I talk about ssf.
33. I selected to develop the framework based on spring, about 5 years ago. This because. Standard DI Container at the time Familiar to many Java Developers Spring has customizable context And SSF provides customized Context for SIP Servlet Many functionality modules are working on spring framework So you can create the converged application with Spring Framework modules
34. This is a sample sip servlet with POJO. @Component annotation Spring will scan this pojo by this annotation. @Autowired annottion Spring will inject objects by this annotation @SipServletRequestMapping annotation SSF call this method by this annotation, when Sip request is received. This is very simple.
35. Of course, you can use AOP function. Let’s add Call-Blocking function to this service.
36. First create a new pojo for call-blocking. This logic is very simple. Determine whether you want to block requests And send response with specified error status.
37. Second Create aspect class for advice This is a simple aspect class with annotations. If proxyhandler.handlerequest is called, this aspect class is called by spring.
38. And setup the configrations for AOP. These configurations are basic settings for spring. Enable the aspectj fucntions And scan classes annotated with @AspectJ annotation.
39. Next Introduce CTF by Jean
40. Contact information Naoki, it’s me. SSF lead Jean MSS Lead Georges CDI Telco framework lead Vladimir Mobicents core developere and high availability support
41. On March 11, 2011, the largest earthquake occurred and tremendous tsunami hit the eastern part of Japan. Immediately after this happened , many countries announced their support. We are in a difficult situation, many disaster victims have no place to live. But we will try to stand up and go on. Many many thanks from Japan Thanks for listening.