Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simulator startup extremely slow (Python CPU utilization 100%) #287

Closed
Suizi11 opened this issue Oct 7, 2020 · 16 comments
Closed

Simulator startup extremely slow (Python CPU utilization 100%) #287

Suizi11 opened this issue Oct 7, 2020 · 16 comments
Assignees
Labels
bug Something isn't working

Comments

@Suizi11
Copy link

Suizi11 commented Oct 7, 2020

For about 6 weeks the simulator startup process is very, very slow. First, the tool tries to login at the container registry, stops the edgeHubDev and removes all the containers from previous runs. After the network check (or whatever this is intended for) the process hangs for about 1 minute. During this time I've one "Python" process in the activity monitor (macOS) which uses 100% of the CPU the whole time. After waiting a minute, everything starts as expected

Login Succeeded
Stopping edgeHubDev ... done
Removing <container>            ... done
Removing <container>            ... done
Removing <container>            ... done
Removing edgeHubDev          ... done
Network azure-iot-edge-dev is external, skipping

--> here anything weird is happening with the process, then it proceeds as always (before)

Pulling edgeHubDev ... done
...
  • $ python3 --version
    Python 3.8.5
  • $ pip3 --version
    pip 20.2.3 from /usr/local/lib/python3.8/site-packages/pip (python 3.8)
  • $ iotedgehubdev --version
    iotedgehubdev, version 0.14.1
  • $ docker --version
    Docker version 19.03.12, build 48a6621
@marianan
Copy link
Collaborator

marianan commented Oct 7, 2020

Thank you filing the issue and sharing the logs - I don't see any issues with the logs you shared. Could you try to install some of the previous releases to see if the performance is consistently better - this will help narrow down any possible regressions.

@marianan marianan self-assigned this Oct 7, 2020
@Suizi11
Copy link
Author

Suizi11 commented Oct 9, 2020

Unfortunately not, python process still at ~100% and startup takes about 3 minutes.

image

I've installed version 0.11.1, because this one isn't available https://pypi.org/project/iotedgehubdev/0.13.0/

pip3 install iotedgehubdev==0.13.0 ERROR: Could not find a version that satisfies the requirement iotedgehubdev==0.13.0 (from versions: 0.1.0, 0.1.1, 0.2.0, 0.3.0, 0.4.0, 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.11.1, 0.14.0, 0.14.1) ERROR: No matching distribution found for iotedgehubdev==0.13.0

@Suizi11
Copy link
Author

Suizi11 commented Oct 19, 2020

Hey @marianan - I think I've found some important additional information for you.

I can only reproduce this behavior if a "Binds" array is provided in the deployment manifest, e.g. my create options look like as following:

{
    "Binds":[
        "<secret-host-source-path>:/mnt/host",
        "/Users/<secret-path>/modules/WebServer/src:/app/src"
    ]
}

If I remove the binds object everything works fine and fast.

@konichi3
Copy link
Collaborator

The issue is in our backlog now. Will circle back when we have an update

@glneo
Copy link

glneo commented May 20, 2021

Looks like it is getting jammed up running the regex here:

match = re.match(EdgeConstants.MOUNT_WIN_REGEX, bind) or re.match(EdgeConstants.MOUNT_LCOW_REGEX, bind)

As I only have the simple path name cases, my temp workaround is to replace that line with:

match = None

causing it to always fallback to the simple token based parsing below.

The proper fix would be to find a better regex that doesn't take 3 minutes to process for each bind.

@marianan marianan added bug Something isn't working and removed InBacklog labels May 25, 2021
@marianan
Copy link
Collaborator

Thank you @glneo the bug is in our backlog.

@Suizi11
Copy link
Author

Suizi11 commented Jun 17, 2021

@marianan - do you've a rough estimation when this issue will be solved or when somebody starts working on it respectively?

Thanks a lot.

@cindydeng1998
Copy link
Contributor

Hi @Suizi11, thank you for sharing your findings with us. We have done some investigation on this issue and have decided to leave the regex parsing @glneo mentioned above as is.

This is because we need to have a thorough way of parsing binds for everyone that uses our tool, even though it is the source of a longer runtime and high CPU usage. The regex pattern here (iotedgehubdev regex) was taken directly from how docker processes binds.

Thank you very much.

@XavierGeerinck
Copy link

Thanks @mhshami01 for referring to this issue. Seeing that the issue is blocking (it just doesn't start the sim anymore) I am wondering what the current proposed workaround is?

@glneo
Copy link

glneo commented Feb 18, 2022

@cindydeng1998 , Even though the regex in question here "was taken directly from how docker processes binds", does not mean it is correct. Docker and this project use different regex engines causing this issue.

As this is still blocking folks like @XavierGeerinck and our team, this issue should not be closed.

@marianan
Copy link
Collaborator

@glneo thank you for your PR. We will review promptly.

@marianan
Copy link
Collaborator

@Suizi11 a new version (0.14.13) of the iotedgehubdev simulator has been released with a fix for the createOptions path array binding. Please try it out and let us know if your original issue still persists.

@ljani
Copy link

ljani commented Mar 18, 2022

0.14.13 fixed the issue for me. However, I had to install that regex dependency manually, not sure why pip didn't install it for me. Though, I forced the iotedgehubdev@0.14.13 install with iotedgedev@3.3.3.

@marianan
Copy link
Collaborator

Thanks @ljani i missed adding regex as a dependency in the setup.py in my earlier change. An update is coming early next week which will include also this fix.

@marianan
Copy link
Collaborator

marianan commented Mar 23, 2022

@ljani the missing regex dependency is now resolved in version 0.14.14. Let us know if you run into other issues.

If you have a few mins we would love to hear your feedback on your developer experience in general at http://aka.ms/edgedevsurvey.

@marianan
Copy link
Collaborator

Closing this issue, please let us know if any of the symptoms persist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

7 participants