背景

运维小伙伴反映一直出现saiku服务器在刷新缓存的时候无法访问的问题因此打算定位一下原因

工具介绍

jstack是java的自带的线程堆栈查看工具

我们可以查看如下

    qixiaobo@qixiaobo.mac.pro:/Users/qixiaobo> jstack -h  17-10-30 22:16     Usage:         jstack [-l] 
    
                  (to connect to running process)         jstack -F [-m] [-l] 
     
                   (to connect to a hung process)         jstack [-m] [-l] 
      
        
       
                     (to connect to a core file)         jstack [-m] [-l] [server_id@]
        
                      (to connect to a remote debug server)     Options:         -F  to force a thread dump. Use when jstack 
         
           does not respond (process is hung)         -m  to print both java and native frames (mixed mode)         -l  long listing. Prints additional information about locks         -h or -help to print this help message 复制代码

通常我们使用jstack pid来进行对应的堆栈打印

实战

执行

    jstack 12345 复制代码

可以看出来当刷新缓存时其他线程执行到获取连接时会在get时等待监视器很明显此时线程在等待RolapSchemaPool相关的资源

经过查看其代码如下

class RolapSchemaPool {        static final Logger LOGGER = Logger.getLogger(RolapSchemaPool.class);        private static final RolapSchemaPool INSTANCE = new RolapSchemaPool();        private final Map
    
     > mapKeyToSchema = new HashMap();        private final Map
     
      > mapMd5ToSchema = new HashMap();             private RolapSchemaPool() {        }             static RolapSchemaPool instance() {            return INSTANCE;        }             synchronized RolapSchema get(String catalogUrl, String connectionKey, String jdbcUser, String dataSourceStr, PropertyList connectInfo) {            return this.get(catalogUrl, connectionKey, jdbcUser, dataSourceStr, (DataSource)null, connectInfo);        }             synchronized RolapSchema get(String catalogUrl, DataSource dataSource, PropertyList connectInfo) {            return this.get(catalogUrl, (String)null, (String)null, (String)null, dataSource, connectInfo);        }             private RolapSchema get(String catalogUrl, String connectionKey, String jdbcUser, String dataSourceStr, DataSource dataSource, PropertyList connectInfo) {            String dialectClassName = connectInfo.get(RolapConnectionProperties.Dialect.name());            String connectionUuidStr = connectInfo.get(RolapConnectionProperties.JdbcConnectionUuid.name());            boolean useSchemaPool = Boolean.parseBoolean(connectInfo.get(RolapConnectionProperties.UseSchemaPool.name(), "true"));            boolean useContentChecksum = Boolean.parseBoolean(connectInfo.get(RolapConnectionProperties.UseContentChecksum.name()));            if (LOGGER.isDebugEnabled()) {                LOGGER.debug("get: catalog=" + catalogUrl + ", connectionKey=" + connectionKey + ", jdbcUser=" + jdbcUser + ", dataSourceStr=" + dataSourceStr + ", dataSource=" + dataSource + ", dialect=" + dialectClassName + ", jdbcConnectionUuid=" + connectionUuidStr + ", useSchemaPool=" + useSchemaPool + ", useContentChecksum=" + useContentChecksum + ", map-size=" + this.mapKeyToSchema.size() + ", md5-map-size=" + this.mapMd5ToSchema.size());            }                 ConnectionKey connectionKey1 = ConnectionKey.create(connectionUuidStr, dataSource, catalogUrl, dialectClassName, connectionKey, jdbcUser, dataSourceStr);            String catalogStr = getSchemaContent(connectInfo, catalogUrl);            SchemaContentKey schemaContentKey = SchemaContentKey.create(connectInfo, catalogUrl, catalogStr);            SchemaKey key = new SchemaKey(schemaContentKey, connectionKey1);            RolapSchema schema = null;            if (!useSchemaPool) {                schema = RolapSchemaLoader.createSchema(key, (ByteString)null, catalogUrl, catalogStr, connectInfo, dataSource);                if (LOGGER.isDebugEnabled()) {                    LOGGER.debug("create (no pool): schema-name=" + schema.name + ", schema-id=" + Integer.toHexString(System.identityHashCode(schema)));                }                     return schema;            } else if (useContentChecksum) {                ByteString md5Bytes = new ByteString(Util.digestMd5(catalogStr));                SoftReference
      
        ref = (SoftReference)this.mapMd5ToSchema.get(md5Bytes);                if (LOGGER.isDebugEnabled()) {                    LOGGER.debug("get(key=" + key + ") returned " + toString(ref));                }                     if (ref != null) {                    schema = (RolapSchema)ref.get();                    if (schema == null) {                        this.mapKeyToSchema.remove(key);                        this.mapMd5ToSchema.remove(md5Bytes);                    }                }                     if (schema == null) {                    schema = RolapSchemaLoader.createSchema(key, md5Bytes, catalogUrl, catalogStr, connectInfo, dataSource);                    if (LOGGER.isDebugEnabled()) {                        LOGGER.debug("create: schema-name=" + schema.name + ", schema-id=" + System.identityHashCode(schema));                    }                         this.putSchema(schema, md5Bytes);                }                     return schema;            } else {                SoftReference
       
         ref = (SoftReference)this.mapKeyToSchema.get(key);                if (LOGGER.isDebugEnabled()) {                    LOGGER.debug("get(key=" + key + ") returned " + toString(ref));                }                     if (ref != null) {                    schema = (RolapSchema)ref.get();                    if (schema == null) {                        this.mapKeyToSchema.remove(key);                    }                }                     if (schema == null) {                    schema = RolapSchemaLoader.createSchema(key, (ByteString)null, catalogUrl, catalogStr, connectInfo, dataSource);                    if (LOGGER.isDebugEnabled()) {                        LOGGER.debug("create: " + schema);                    }                         this.putSchema(schema, (ByteString)null);                }                     return schema;            }        }复制代码

很明显此处getConnection是synchronized方法而其他用户登录因此也需要等待缓存刷新的线程释放才可以。

经过查看该版本的jar是

实质上其他版本的mondrian并非是同步版本的比如

很明显该版本并非同步版本的代码

spark版本当刷新缓存时对应需要加同步

    - locked <0x00000007f0f0a438> (a com.mysql.jdbc.JDBC4Connection)             - locked <0x00000007ea451028> (a com.mysql.jdbc.PreparedStatement)             at com.mysql.jdbc.PreparedStatement.getMetaData(PreparedStatement.java:2890)             - locked <0x00000007ea44f0a0> (a com.mysql.jdbc.JDBC4PreparedStatement)             at org.apache.commons.dbcp.DelegatingPreparedStatement.getMetaData(DelegatingPreparedStatement.java:193)             at org.apache.commons.dbcp.DelegatingPreparedStatement.getMetaData(DelegatingPreparedStatement.java:193)             at mondrian.rolap.RolapSchema$PhysSchema.describe(RolapSchema.java:1017)             at mondrian.rolap.RolapSchema$PhysCalcColumn.compute(RolapSchema.java:2299)             at mondrian.rolap.RolapSchemaLoader.registerColumn(RolapSchemaLoader.java:1422)             at mondrian.rolap.RolapSchemaLoader.registerTable(RolapSchemaLoader.java:1057)             at mondrian.rolap.RolapSchemaLoader.validatePhysicalSchema(RolapSchemaLoader.java:715)             at mondrian.rolap.RolapSchemaLoader.loadStage2(RolapSchemaLoader.java:376)             at mondrian.rolap.RolapSchemaLoader.loadStage1(RolapSchemaLoader.java:336)             at mondrian.rolap.RolapSchemaLoader.loadStage0(RolapSchemaLoader.java:272)             at mondrian.rolap.RolapSchemaLoader.createSchema(RolapSchemaLoader.java:4305)             at mondrian.rolap.RolapSchemaPool.get(RolapSchemaPool.java:210)             at mondrian.rolap.RolapSchemaPool.get(RolapSchemaPool.java:62)             - locked <0x000000070505d428> (a mondrian.rolap.RolapSchemaPool)             at mondrian.rolap.RolapConnection.
    
     (RolapConnection.java:160)             at mondrian.rolap.RolapConnection.
     
      (RolapConnection.java:84)             at mondrian.olap.DriverManager.getConnection(DriverManager.java:112)             at mondrian.olap.DriverManager.getConnection(DriverManager.java:68)             at mondrian.olap4j.MondrianOlap4jConnection.
      
       (MondrianOlap4jConnection.java:153)             at mondrian.olap4j.FactoryJdbc4Plus$AbstractConnection.
       
        (FactoryJdbc4Plus.java:323)             at mondrian.olap4j.FactoryJdbc41Impl$MondrianOlap4jConnectionJdbc41.
        
         (FactoryJdbc41Impl.java:118)             at mondrian.olap4j.FactoryJdbc41Impl.newConnection(FactoryJdbc41Impl.java:32)             at mondrian.olap4j.MondrianOlap4jDriver.connect(MondrianOlap4jDriver.java:139)             at java.sql.DriverManager.getConnection(DriverManager.java:571)             at java.sql.DriverManager.getConnection(DriverManager.java:215)             at org.saiku.datasources.connection.SaikuOlapConnection.connect(SaikuOlapConnection.java:107)             at org.saiku.datasources.connection.SaikuOlapConnection.connect(SaikuOlapConnection.java:56)             at org.saiku.datasources.connection.SaikuConnectionFactory.getConnection(SaikuConnectionFactory.java:29)             at org.saiku.web.core.SecurityAwareConnectionManager.connect(SecurityAwareConnectionManager.java:293)             at org.saiku.web.core.SecurityAwareConnectionManager.getInternalConnection(SecurityAwareConnectionManager.java:109)             at org.saiku.web.core.SecurityAwareConnectionManager.refreshInternalConnection(SecurityAwareConnectionManager.java:146)             at org.saiku.datasources.connection.AbstractConnectionManager.refreshConnection(AbstractConnectionManager.java:151)             at org.saiku.service.olap.OlapDiscoverService.refreshConnection(OlapDiscoverService.java:87)             at org.saiku.web.rest.resources.OlapDiscoverResource.refreshConnection(OlapDiscoverResource.java:115)             at sun.reflect.GeneratedMethodAccessor308.invoke(Unknown Source)             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 复制代码

我们没有使用spark是否可以考虑使用略低版本的mondrian呢？