Too many open files - Failed to initialize inotify: the user limit on the total number of inotify instances has been reached

env

• rhel6.9
• nacos
• jdk1.8.x

问题

16:21:41.834 [main] ERROR org.springframework.boot.SpringApplication - Application run failed
com.alibaba.nacos.api.exception.runtime.NacosRuntimeException: ErrCode:500, ErrMsg:User limit of inotify instances reached or too many open files
	at com.alibaba.nacos.core.listener.StartingApplicationListener.loadPreProperties(StartingApplicationListener.java:142)
    ...
Caused by: com.alibaba.nacos.api.exception.NacosException: java.io.IOException: User limit of inotify instances reached or too many open files
	at com.alibaba.nacos.sys.file.WatchFileCenter$WatchDirJob.<init>(WatchFileCenter.java:189)
	at com.alibaba.nacos.sys.file.WatchFileCenter.registerWatcher(WatchFileCenter.java:97)
	at com.alibaba.nacos.core.listener.StartingApplicationListener.registerWatcher(StartingApplicationListener.java:148)
	at com.alibaba.nacos.core.listener.StartingApplicationListener.loadPreProperties(StartingApplicationListener.java:140)
	... 16 common frames omitted
Caused by: java.io.IOException: User limit of inotify instances reached or too many open files
	at sun.nio.fs.LinuxWatchService.<init>(LinuxWatchService.java:64)
	at sun.nio.fs.LinuxFileSystem.newWatchService(LinuxFileSystem.java:47)
	at com.alibaba.nacos.sys.file.WatchFileCenter$WatchDirJob.<init>(WatchFileCenter.java:184)
	... 19 common frames omitted

分析

ulimit -n    #查看当前用户默认的最大文件句柄数
lsof |wc -l  #查看所有进程的文件打开数

lsof -n|awk '{print $2}'|sort|uniq -c|sort -nr|more
查看当前进程打开了多少句柄数，注意第一列是句柄数，第二列是进程id这样可以通过进程ID确定具体操作系统句柄数效果的进行是哪个从而进一步分析问题

echo "fs.inotify.max_user_watches=524288" | sudo tee -a /etc/sysctl.conf
echo "fs.inotify.max_user_instances=8192" | sudo tee -a /etc/sysctl.conf

sysctl -p /etc/sysctl.conf

解决

cat /proc/sys/fs/inotify/max_user_instances   #default 128

m1:（重启失效）
sysctl -w fs.inotify.max_user_instances=256

m2:
/etc/sysctl.conf
fs.inotify.max_user_instances = 256


//查看系统限制最大打开文件数
sysctl fs.file-max 
    fs.file-max = 1208702

sysctl -w fs.file-max=<NEWVALUE>

//Too many Open Files
/etc/security/limits.conf
*  soft    nofile  1000000
*  hard    nofile  1000000

ulimit -n