ActiveMQ - Large number of queues(HowTo)

Large number of queues(HowTo)

http://activemq.2283324.n4.nabble.com/Large-number-of-queues-HowTo-td2364929.html#a2364929

How do I configure 10s of 1000s of Queues in a single broker ?

http://activemq.apache.org/how-do-i-configure-10s-of-1000s-of-queues-in-a-single-broker-.html

able to create more than 20k queues using ~30 threads in a test environment with a 512MB heap. (by Jose Luna-2)

<Reduce Threads>

많은 수의 queue를 만들면 다음과 같은 에러를 만나게 된다.

"unable to create new native thread"

이것은 QueueThread가 queue당 하나가 생성되기 때문이다.

http://activemq.apache.org/per-destination-policies.html 에서 queue properties를 설정할 수 있는데,

optimizedDispatch=true(기본값은 false)로 설정하면 된다.

이 옵션은 다음과 같이 설명하고 있다. "don't use a separate thread for dispatching from a Queue"

그러나, 실제로는 queue별로 thread가 생성된다. 이는 optimizedDispatch 는 TaskRunner를 사용하는 thread를 생성하기 때문이다.

thread pool을 사용하기 위해서는 org.apache.activemq.UseDedicatedTaskRunner 를 false로 설정해 줘야 한다.

http://activemq.apache.org/javalangoutofmemory.html - Number of Threads 항목참조

Number of Threads

By default, ActiveMQ uses a dedicated thread per destination. If there are large numbers of Destinations there will be a large number of threads and their associated memory resource usage. ActiveMQ can be configured to use a thread pool through the use of the system property: -Dorg.apache.activemq.UseDedicatedTaskRunner=false. This is currently specified in the activemq start script via ACTIVEMQ_OPTS. Using a thread pool can restrict the number of threads required by ActiveMQ and hence reduce memory usage.

<policyEntries> 

    <policyEntry queue=">" memoryLimit="15mb" optimizedDispatch='true'/> 

</policyEntries>

set the ACTIVEMQ_OPTS to disable dedicated task runners in the start up script, INSTALL_DIR/bin/activemq -e.g.

   ACTIVEMQ_OPTS="-Xmx512M -Dorg.apache.activemq.UseDedicatedTaskRunner=false"  

or you can set ACTIVEMQ_OPTS in /etc/activemq.conf.

<Reduce number of file descriptors>

AMQ Message Store를 사용하기 보다 KahaDB를 사용하기를 권장함(in version 5.3 higher)

※ AMQ Message Store는 queue당 file descriptor 하나를 사용함.

<Reducing Memory Consumption>

Reduce the memory used per thread - http://activemq.apache.org/javalangoutofmemory.html

queue properties

http://activemq.apache.org/per-destination-policies.html

How do I configure 10s of 1000s of Queues in a single broker ?

http://activemq.apache.org/how-do-i-configure-10s-of-1000s-of-queues-in-a-single-broker-.html

memory exceptions

http://activemq.apache.org/javalangoutofmemory.html

Understanding the Threads Allocated in ActiveMQ

http://fusesource.com/wiki/display/ProdInfo/Understanding+the+Threads+Allocated+in+ActiveMQ