发表新帖

swoole不兼容grpc扩展，开启后会导致server堵住

### 问题描述
 swoole不兼容grpc扩展，开启后会导致程序堵住

### Swoole版本，PHP版本，以及操作系统版本信息

[root@localhost no-debug-non-zts-20180731]# php -version
PHP 7.3.28 (cli) (built: Jul 19 2021 21:18:26) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.28, Copyright (c) 1998-2018 Zend Technologies
    with the ionCube PHP Loader + ionCube24 v10.4.5, Copyright (c) 2002-2020, by ionCube Ltd.
    with Zend OPcache v7.3.28, Copyright (c) 1999-2018, by Zend Technologies

[root@localhost no-debug-non-zts-20180731]# php --ri swoole

swoole

Swoole => enabled
Author => Swoole Team <team@swoole.com>
Version => 4.7.1
Built => Oct 22 2021 00:26:11
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
openssl => OpenSSL 1.1.1k  25 Mar 2021
dtls => enabled
pcre => enabled
zlib => 1.2.11
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608
[root@localhost no-debug-non-zts-20180731]# php --ri grpc

grpc

grpc support => enabled
grpc module version => 1.38.0

Directive => Local Value => Master Value
grpc.enable_fork_support => 0 => 0
grpc.poll_strategy => no value => no value
grpc.grpc_verbosity => no value => no value
grpc.grpc_trace => no value => no value
grpc.log_filename => no value => no value

### 相关代码
```php

$this->serv = new swoole_http_server($conf['host'], $conf['port']);
        $this->serv->set([
            'worker_num' => $conf['worker_num'] ?? 2, // worker进程数量
            'max_request' => $conf['max_request'], // 最大请求次数，当请求大于它时，将会自动重启该worker
            'daemonize' => $conf['daemonize'] != '' ? $conf['daemonize'] : ($conf['env'] == 'development' ? false : true),
            // 'dispatch_mode' => 1,
            'log_file' => $conf['log_file'],
            'enable_static_handler' => true,
            'document_root' => dirname(dirname(__DIR__)) . '/public',
            'static_handler_locations' => ['/static', '/assets'],
            'enable_reuse_port' => $conf['enable_reuse_port'] ?? false,
            // 'hook_flags'=> SWOOLE_HOOK_ALL // v4.4+版本使用此方法。
        ]);
```

### 你期待的结果是什么？实际看到的错误信息又是什么？

http server正常工作

实际wget无响应

```
#0  0x00007f0b641701c9 in syscall () from /lib64/libc.so.6
#1  0x00007f0b59113d4e in WaitUntil (t=..., val=0, v=0x7f0b6a33c140) at /home/code/larastack/build/src/grpc-1.38.0/third_party/abseil-cpp/absl/synchronization/internal/futex.h:104
#2  absl::lts_20210324::synchronization_internal::Waiter::Wait (this=this@entry=0x7f0b6a33c140, t=t@entry=...) at /home/code/larastack/build/src/grpc-1.38.0/third_party/abseil-cpp/absl/synchronization/internal/waiter.cc:95
#3  0x00007f0b59113c61 in AbslInternalPerThreadSemWait_lts_20210324 (t=...) at /home/code/larastack/build/src/grpc-1.38.0/third_party/abseil-cpp/absl/synchronization/internal/per_thread_sem.cc:93
#4  0x00007f0b59117b7b in absl::lts_20210324::CondVar::WaitCommon (this=this@entry=0x7f0b596069f8 <g_cv_shutdown>, mutex=mutex@entry=0x7f0b59606a10 <_ZL4g_mu>, t=t@entry=...) at /home/code/larastack/build/src/grpc-1.38.0/third_party/abseil-cpp/absl/synchronization/mutex.cc:2559
#5  0x00007f0b59117d17 in absl::lts_20210324::CondVar::Wait (this=this@entry=0x7f0b596069f8 <g_cv_shutdown>, mu=mu@entry=0x7f0b59606a10 <_ZL4g_mu>) at /home/code/larastack/build/src/grpc-1.38.0/third_party/abseil-cpp/absl/synchronization/mutex.cc:2594
#6  0x00007f0b58ffcce3 in gpr_cv_wait (cv=cv@entry=0x7f0b596069f8 <g_cv_shutdown>, mu=mu@entry=0x7f0b59606a10 <_ZL4g_mu>, abs_deadline=...) at /home/code/larastack/build/src/grpc-1.38.0/src/core/lib/gpr/sync_abseil.cc:84
#7  0x00007f0b59029f4b in stop_threads () at /home/code/larastack/build/src/grpc-1.38.0/src/core/lib/iomgr/timer_manager.cc:326
#8  0x00007f0b5902aa9a in grpc_timer_manager_set_threading (enabled=enabled@entry=false) at /home/code/larastack/build/src/grpc-1.38.0/src/core/lib/iomgr/timer_manager.cc:349
#9  0x00007f0b590a2567 in grpc_shutdown_internal_locked () at /home/code/larastack/build/src/grpc-1.38.0/src/core/lib/surface/init.cc:177
#10 0x00007f0b590a2838 in grpc_shutdown () at /home/code/larastack/build/src/grpc-1.38.0/src/core/lib/surface/init.cc:228
#11 0x00007f0b590cefe3 in zm_shutdown_grpc (type=<optimized out>, module_number=<optimized out>) at /home/code/larastack/build/src/grpc-1.38.0/src/php/ext/grpc/php_grpc.c:568
#12 0x00000000008ea107 in module_destructor (module=module@entry=0x2291d50) at /home/code/larastack/build/src/php-7.3.28/Zend/zend_API.c:2575
#13 0x00000000008e3e3c in module_destructor_zval (zv=<optimized out>) at /home/code/larastack/build/src/php-7.3.28/Zend/zend.c:745
#14 0x00000000008f44c0 in _zend_hash_del_el_ex (prev=<optimized out>, p=<optimized out>, idx=52, ht=<optimized out>) at /home/code/larastack/build/src/php-7.3.28/Zend/zend_hash.c:1182
#15 _zend_hash_del_el (p=0x224d780, idx=52, ht=0x14f8ee0 <module_registry>) at /home/code/larastack/build/src/php-7.3.28/Zend/zend_hash.c:1205
#16 zend_hash_graceful_reverse_destroy (ht=ht@entry=0x14f8ee0 <module_registry>) at /home/code/larastack/build/src/php-7.3.28/Zend/zend_hash.c:1659
#17 0x00000000008e850c in zend_destroy_modules () at /home/code/larastack/build/src/php-7.3.28/Zend/zend_API.c:2013
#18 0x00000000008e4ad3 in zend_shutdown () at /home/code/larastack/build/src/php-7.3.28/Zend/zend.c:976
#19 0x0000000000888572 in php_module_shutdown () at /home/code/larastack/build/src/php-7.3.28/main/main.c:2496
#20 0x0000000000460260 in main (argc=2, argv=0x21ec730) at /home/code/larastack/build/src/php-7.3.28/sapi/cli/php_cli.c:1411
```

发布于4年前 · 40 次浏览 · 来自提问

ganl

### 问题描述
 swoole不兼容grpc扩展，开启后会导致程序堵住

### Swoole版本，PHP版本，以及操作系统版本信息

[root@localhost no-debug-non-zts-20180731]# php --ri swoole

swoole

grpc

grpc support => enabled
grpc module version => 1.38.0

### 相关代码
```php

### 你期待的结果是什么？实际看到的错误信息又是什么？

http server正常工作

实际wget无响应

登录后参与评论

2021-10-25

ganl

2021-10-25

序冢

2021-11-10

ganl

dump 掉了

```bash
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `i2up_sw:primary                                                              '.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f7cf22eb674 in absl::lts_20210324::CondVar::Wakeup (this=0x7f7cf27dba00 <g_cv_wait>, w=0x7f7d036d0300)
    at /home/code/larastack/build/src/grpc-1.38.0/third_party/abseil-cpp/absl/synchronization/mutex.cc:2602
2602    /home/code/larastack/build/src/grpc-1.38.0/third_party/abseil-cpp/absl/synchronization/mutex.cc: No such file or directory.
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.el7.x86_64 glibc-2.17-292.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_7.2.x86_64 libcom_err-1.42.9-16.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libicu-50.2-3.el7.x86_64 libjpeg-turbo-1.2.90-8.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 libselinux-2.5-14.1.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64 nspr-4.21.0-1.el7.x86_64 nss-3.44.0-4.el7.x86_64 nss-softokn-freebl-3.44.0-5.el7.x86_64 nss-util-3.44.0-3.el7.x86_64 openldap-2.4.44-21.el7_6.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 pcre-8.32-17.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64
(gdb) bt
#0  0x00007f7cf22eb674 in absl::lts_20210324::CondVar::Wakeup (this=0x7f7cf27dba00 <g_cv_wait>, w=0x7f7d036d0300)
    at /home/code/larastack/build/src/grpc-1.38.0/third_party/abseil-cpp/absl/synchronization/mutex.cc:2602
#1  0x00007f7cf22eb7d1 in absl::lts_20210324::CondVar::SignalAll (this=this@entry=0x7f7cf27dba00 <g_cv_wait>)
    at /home/code/larastack/build/src/grpc-1.38.0/third_party/abseil-cpp/absl/synchronization/mutex.cc:2675
#2  0x00007f7cf21d1d15 in gpr_cv_broadcast (cv=cv@entry=0x7f7cf27dba00 <g_cv_wait>) at /home/code/larastack/build/src/grpc-1.38.0/src/core/lib/gpr/sync_abseil.cc:102
#3  0x00007f7cf21fef03 in stop_threads () at /home/code/larastack/build/src/grpc-1.38.0/src/core/lib/iomgr/timer_manager.cc:321
#4  0x00007f7cf21ffa9a in grpc_timer_manager_set_threading (enabled=enabled@entry=false) at /home/code/larastack/build/src/grpc-1.38.0/src/core/lib/iomgr/timer_manager.cc:349
#5  0x00007f7cf2277567 in grpc_shutdown_internal_locked () at /home/code/larastack/build/src/grpc-1.38.0/src/core/lib/surface/init.cc:177
#6  0x00007f7cf2277838 in grpc_shutdown () at /home/code/larastack/build/src/grpc-1.38.0/src/core/lib/surface/init.cc:228
#7  0x00007f7cf22a3fe3 in zm_shutdown_grpc (type=<optimized out>, module_number=<optimized out>) at /home/code/larastack/build/src/grpc-1.38.0/src/php/ext/grpc/php_grpc.c:568
#8  0x00000000008e2cb7 in module_destructor (module=module@entry=0x17db450) at /home/code/larastack/build/src/php-7.3.31/Zend/zend_API.c:2575
#9  0x00000000008dbe6c in module_destructor_zval (zv=<optimized out>) at /home/code/larastack/build/src/php-7.3.31/Zend/zend.c:745
#10 0x00000000008ecd18 in _zend_hash_del_el_ex (prev=<optimized out>, p=<optimized out>, idx=52, ht=<optimized out>) at /home/code/larastack/build/src/php-7.3.31/Zend/zend_hash.c:1182
#11 _zend_hash_del_el (p=0x1796920, idx=52, ht=0x14e03e0 <module_registry>) at /home/code/larastack/build/src/php-7.3.31/Zend/zend_hash.c:1205
#12 zend_hash_graceful_reverse_destroy (ht=ht@entry=0x14e03e0 <module_registry>) at /home/code/larastack/build/src/php-7.3.31/Zend/zend_hash.c:1659
#13 0x00000000008e10bc in zend_destroy_modules () at /home/code/larastack/build/src/php-7.3.31/Zend/zend_API.c:2013
#14 0x00000000008dd6d3 in zend_shutdown () at /home/code/larastack/build/src/php-7.3.31/Zend/zend.c:976
#15 0x0000000000881302 in php_module_shutdown () at /home/code/larastack/build/src/php-7.3.31/main/main.c:2496
#16 0x0000000000461c6a in main (argc=6, argv=0x17352e0) at /home/code/larastack/build/src/php-7.3.31/sapi/cli/php_cli.c:1411
(gdb)

```

2021-11-14

ganl

swoole 4.8.1
grpc 1.41.0

两个拓展默认配置，在不开启Opcache的情况下，kill -TERM 会有dump产生
```bash

munmap(0x7f8fb8414000, 8392704)         = 0
sendto(21, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\17\0\0\0\0\0\0\0\0\0"..., 40, 0, NULL, 0) = 40
futex(0x7f8fa75fc9d0, FUTEX_WAIT, 97332, NULL) = -1 EAGAIN (Resource temporarily unavailable)
munmap(0x7f8fb8c15000, 8392704)         = 0
sendto(23, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\17\0\0\0\0\0\0\0\0\0"..., 40, 0, NULL, 0) = 40
futex(0x7f8f9ffff9d0, FUTEX_WAIT, 97333, NULL) = 0
munmap(0x7f8fb9416000, 8392704)         = 0
sendto(25, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\17\0\0\0\0\0\0\0\0\0"..., 40, 0, NULL, 0) = 40
futex(0x7f8fa6dfb9d0, FUTEX_WAIT, 97334, NULL) = -1 EAGAIN (Resource temporarily unavailable)
munmap(0x7f8fa7dfe000, 8392704)         = 0
close(5)                                = 0
rt_sigprocmask(SIG_UNBLOCK, [HUP USR1 USR2 PIPE TERM CHLD VTALRM RT_2], NULL, 8) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=97326, si_uid=0, si_status=0, si_utime=0, si_stime=1} ---
close(28)                               = 0
write(1, "[2021-11-14 01:20:22]: server is"..., 42) = 42
munmap(0x7f8fa960e000, 27200008)        = 0
munmap(0x7f8fa64c0000, 430080)          = 0
munmap(0x7f8fa6529000, 430080)          = 0
munmap(0x7f8fa6592000, 430080)          = 0
munmap(0x7f8fa6457000, 430080)          = 0
munmap(0x7f8fc4800000, 430080)          = 0
munmap(0x7f8fc4869000, 430080)          = 0
munmap(0x7f8fcb4ff000, 430080)          = 0
munmap(0x7f8fc48d2000, 430080)          = 0
munmap(0x7f8fcb637000, 73824)           = 0
munmap(0x7f8faafff000, 16777224)        = 0
munmap(0x7f8fcb691000, 80)              = 0
munmap(0x7f8fcb690000, 1224)            = 0
close(12)                               = 0
close(11)                               = 0
close(14)                               = 0
close(13)                               = 0
close(16)                               = 0
close(15)                               = 0
close(18)                               = 0
close(17)                               = 0
close(20)                               = 0
close(19)                               = 0
close(22)                               = 0
close(21)                               = 0
close(24)                               = 0
close(23)                               = 0
close(26)                               = 0
close(25)                               = 0
close(0)                                = 0
munmap(0x7f8fa8800000, 2097152)         = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x7f8fb8c13af0} ---
+++ killed by SIGSEGV (core dumped) +++

```

解决：
https://github.com/grpc/grpc/issues/18833#issuecomment-493541026
https://github.com/grpc/grpc/pull/18539

; Enable "General RPC framework" extension module
extension=grpc.so

; Configuration
**grpc.enable_fork_support = 1**
;grpc.poll_strategy = ''
;grpc.grpc_verbosity = ''
;grpc.grpc_trace = ''
;grpc.log_filename = ''

```bash
strace: Process 69455 attached
wait4(-1, 0x7ffc785adf38, 0, NULL)      = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=69445, si_uid=0} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
alarm(6)                                = 0
kill(69473, SIGTERM)                    = 0
kill(69480, SIGTERM)                    = 0
kill(69493, SIGTERM)                    = 0
kill(69500, SIGTERM)                    = 0
--- SIGIO {si_signo=SIGIO, si_code=SI_USER, si_pid=69473, si_uid=0} ---
rt_sigreturn({mask=[]})                 = 0
kill(69513, SIGTERM)                    = 0
kill(69520, SIGTERM)                    = 0
kill(69529, SIGTERM)                    = 0
kill(69540, SIGTERM)                    = 0
wait4(69473, 0x7ffc785ade6c, 0, NULL)   = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGIO {si_signo=SIGIO, si_code=SI_USER, si_pid=69540, si_uid=0} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
wait4(69473, 0x7ffc785ade6c, 0, NULL)   = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=69513, si_uid=0, si_status=0, si_utime=193, si_stime=23} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
wait4(69473, 0x7ffc785ade6c, 0, NULL)   = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=69529, si_uid=0, si_status=0, si_utime=212, si_stime=27} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
wait4(69473, 0x7ffc785ade6c, 0, NULL)   = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=69500, si_uid=0, si_status=SIGTERM, si_utime=179, si_stime=25} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
wait4(69473, 0x7ffc785ade6c, 0, NULL)   = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=69480, si_uid=0, si_status=0, si_utime=184, si_stime=25} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
wait4(69473, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 69473
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=69473, si_uid=0, si_status=0, si_utime=191, si_stime=24} ---
rt_sigreturn({mask=[]})                 = 69473
wait4(69480, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 69480
wait4(69493, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGTERM}], 0, NULL) = 69493
wait4(69500, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGTERM}], 0, NULL) = 69500
wait4(69513, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 69513
wait4(69520, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 69520
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=69520, si_uid=0, si_status=0, si_utime=237, si_stime=27} ---
rt_sigreturn({mask=[]})                 = 69520
wait4(69529, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 69529
wait4(69540, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 69540
alarm(0)                                = 6
exit_group(0)                           = ?
+++ exited with 0 +++

```

```bash
setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
sendto(19, "\301\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\4\0\16\0\0\0\0\0\0\0\0\0"..., 40, 0, NULL, 0) = 40
accept4(5, {sa_family=AF_INET, sin_port=htons(47638), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_CLOEXEC|SOCK_NONBLOCK) = 93
setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
sendto(21, "\302\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\16\0\0\0\0\0\0\0\0\0"..., 40, 0, NULL, 0) = 40
accept4(5, 0x37ae5f4, 0x37ae664, SOCK_CLOEXEC|SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
epoll_wait(27, [], 4096, 379)           = 0
epoll_wait(27, [], 4096, 999)           = 0
epoll_wait(27, [], 4096, 996)           = 0
epoll_wait(27, [], 4096, 999)           = 0
epoll_wait(27, [], 4096, 999)           = 0
epoll_wait(27, [{EPOLLIN, {u32=58343840, u64=58343840}}], 4096, 999) = 1
accept4(5, {sa_family=AF_INET, sin_port=htons(48152), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_CLOEXEC|SOCK_NONBLOCK) = 37
setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
sendto(21, "\303\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\16\0\0\0\0\0\0\0\0\0"..., 40, 0, NULL, 0) = 40
accept4(5, {sa_family=AF_INET, sin_port=htons(48154), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_CLOEXEC|SOCK_NONBLOCK) = 38
setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
sendto(23, "\304\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\0\16\0\0\0\0\0\0\0\0\0"..., 40, 0, NULL, 0) = 40
accept4(5, 0x37aab84, 0x37aabf4, SOCK_CLOEXEC|SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
epoll_wait(27,  <unfinished ...>
+++ killed by SIGTERM +++

```

`dmesg  |grep grpc` 进一步确认没有dump

ps: grpc 1.34以下Worker进程重启,卡住阻塞：
```bash
gpr_cv_wait
grpc_shutdown_internal_locked
grpc_shutdown_blocking
```

总结：
关闭cli模式的Opcache，
grpc拓展增加配置`grpc.enable_fork_support = 1`
暂时没发现其他问题。